{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| [06_text_generation/01_字符级人名生成.ipynb](https://github.com/shibing624/nlp-tutorial/tree/main/06_text_generation/01_字符级人名生成.ipynb)  | 从头训练Char-RNN做人名生成  |[Open In Colab](https://colab.research.google.com/github/shibing624/nlp-tutorial/blob/main/06_text_generation/01_字符级人名生成.ipynb) |\n",
    "\n",
    "\n",
    "# 基与字符级RNN（Char-RNN）的人名生成\n",
    "\n",
    "**作者**: [Sean Robertson](https://github.com/spro/practical-pytorch)\n",
    "\n",
    "在上一个教程 [姓名识别国籍](https://github.com/shibing624/nlp-tutorial/blob/main/04_text_classification/04_%E5%BA%94%E7%94%A8_%E5%A7%93%E5%90%8D%E8%AF%86%E5%88%AB%E5%9B%BD%E7%B1%8D.ipynb)\n",
    "里我们使用RNN把名字分类到它所属的语言中, 这次我们改变一下来学习从语言中生成名字. \n",
    "\n",
    "::\n",
    "\n",
    "    > python sample.py Russian RUS\n",
    "    Rovakov\n",
    "    Uantov\n",
    "    Shavakov\n",
    "\n",
    "    > python sample.py German GER\n",
    "    Gerren\n",
    "    Ereng\n",
    "    Rosher\n",
    "\n",
    "    > python sample.py Spanish SPA\n",
    "    Salla\n",
    "    Parer\n",
    "    Allan\n",
    "\n",
    "    > python sample.py Chinese CHI\n",
    "    Chan\n",
    "    Hang\n",
    "    Iun\n",
    "\n",
    "我们仍然手工搭建一个包含几个线性层的小的RNN. 这次的最大的不同是输入一个类别, 每次输出一个字母, \n",
    "而不是读入所有名字的字母来预测一个类别. 循环的预测每一个字母来构成语言（也可以用文\n",
    "字或者其他更高级的结构完成）, 通常被称为“语言模型”. \n",
    "\n",
    "**推荐阅读: **\n",
    "\n",
    "假设你至少安装了PyTorch, 熟悉Python, 理解Tensors: \n",
    "\n",
    "-  http://pytorch.org/ : 安装说明\n",
    "-  :doc:`/beginner/deep_learning_60min_blitz` 获取一般的 PyTorch 入门\n",
    "-  :doc:`/beginner/pytorch_with_examples` 广泛且深入的概述\n",
    "-  :doc:`/beginner/former_torchies_tutorial` 如果曾经是 Lua Torch 的用户\n",
    "\n",
    "下面这些对了解 RNNs 和其工作原理也是很有用的: \n",
    "\n",
    "-  `The Unreasonable Effectiveness of Recurrent Neural\n",
    "   Networks <http://karpathy.github.io/2015/05/21/rnn-effectiveness/>`\n",
    "   展示了一系列真实生活中的例子\n",
    "-  `Understanding LSTM\n",
    "   Networks <http://colah.github.io/posts/2015-08-Understanding-LSTMs/>`\n",
    "   是一篇特别关于LSTMs的文章, 但是对于一般的RNNs也很有益的\n",
    "\n",
    "\n",
    "\n",
    "## 数据准备\n",
    "\n",
    "\n",
    "从 `这里 <https://download.pytorch.org/tutorial/data.zip>`下载数据, 并解压到当前目录. (本教程已经下载过了)\n",
    "\n",
    "更多的细节参考上一个教程, 总之, 数据含有一批纯文本文件:  ``data/names/[Language].txt`` \n",
    "每一行一个人名. 将行分割成数组, 并把 Unicode 转换成 ASCII 编码, 最后放进一个字典里 ``{language: [names ...]}``. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# categories: 18 ['Czech', 'German', 'Arabic', 'Japanese', 'Chinese', 'Vietnamese', 'Russian', 'French', 'Irish', 'English', 'Spanish', 'Greek', 'Italian', 'Portuguese', 'Scottish', 'Dutch', 'Korean', 'Polish']\n",
      "O'Neal\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from io import open\n",
    "import glob\n",
    "import unicodedata\n",
    "import string\n",
    "\n",
    "all_letters = string.ascii_letters + \" .,;'-\"\n",
    "n_letters = len(all_letters) + 1 # 添加 EOS 标记\n",
    "\n",
    "def findFiles(path): return glob.glob(path)\n",
    "\n",
    "# 将 Unicode 字符串转换为纯 ASCII 编码, 感谢 http://stackoverflow.com/a/518232/2809427\n",
    "def unicodeToAscii(s):\n",
    "    return ''.join(\n",
    "        c for c in unicodedata.normalize('NFD', s)\n",
    "        if unicodedata.category(c) != 'Mn'\n",
    "        and c in all_letters\n",
    "    )\n",
    "\n",
    "# 读取文件并分割成行\n",
    "def readLines(filename):\n",
    "    with open(filename, encoding='utf-8') as f:\n",
    "        return [unicodeToAscii(line.strip()) for line in f]\n",
    "\n",
    "# 构建映射字典 category_lines , 每个类别是由很多个行组成的list\n",
    "category_lines = {}\n",
    "all_categories = []\n",
    "for filename in findFiles('data/names/*.txt'):\n",
    "    category = os.path.splitext(os.path.basename(filename))[0]\n",
    "    all_categories.append(category)\n",
    "    lines = readLines(filename)\n",
    "    category_lines[category] = lines\n",
    "\n",
    "n_categories = len(all_categories)\n",
    "\n",
    "print('# categories:', n_categories, all_categories)\n",
    "print(unicodeToAscii(\"O'Néàl\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 创建网络\n",
    "\n",
    "这个网络扩展了RNN网络, 为类别张量添加了一个额外的参数, 并和其他的参数串联在一起. 类别张量\n",
    "和字母的输入一样是 one-hot 向量. \n",
    "\n",
    "我们将输出解释成为下一个字母的概率, 采样的时候, 最有可能的输出被当做下一个输入. \n",
    "\n",
    "为了让网络更加有效工作, 我添加了第二个线性层 ``o2o`` （在合并了隐藏层和输出层的后面）. \n",
    "还有一个 Dropout 层,`使输入的部分值以给定的概率值随机的变成0` [dropout paper](https://arxiv.org/abs/1207.0580) \n",
    "（这里概率取0.1）, 这样做通常是为了模糊输入以防止过拟合. 这里我们在网络的最末端使用它, 从而故意添加一些混乱和增加采样的多样化. \n",
    "\n",
    "![net](https://i.imgur.com/Z2xbySO.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "class RNN(nn.Module):\n",
    "    def __init__(self, input_size, hidden_size, output_size):\n",
    "        super(RNN, self).__init__()\n",
    "        self.hidden_size = hidden_size\n",
    "\n",
    "        self.i2h = nn.Linear(n_categories + input_size + hidden_size, hidden_size)\n",
    "        self.i2o = nn.Linear(n_categories + input_size + hidden_size, output_size)\n",
    "        self.o2o = nn.Linear(hidden_size + output_size, output_size)\n",
    "        self.dropout = nn.Dropout(0.1)\n",
    "        self.softmax = nn.LogSoftmax(dim=1)\n",
    "\n",
    "    def forward(self, category, input, hidden):\n",
    "        input_combined = torch.cat((category, input, hidden), 1)\n",
    "        hidden = self.i2h(input_combined)\n",
    "        output = self.i2o(input_combined)\n",
    "        output_combined = torch.cat((hidden, output), 1)\n",
    "        output = self.o2o(output_combined)\n",
    "        output = self.dropout(output)\n",
    "        output = self.softmax(output)\n",
    "        return output, hidden\n",
    "\n",
    "    def initHidden(self):\n",
    "        return torch.zeros(1, self.hidden_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练\n",
    "\n",
    "### 训练前的准备\n",
    "\n",
    "首先, 利用辅助函数产生随机的（category, line）对: \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "# 从list中随机选取项\n",
    "def randomChoice(l):\n",
    "    return l[random.randint(0, len(l) - 1)]\n",
    "\n",
    "# 获取随机的类别和该类别中随机的行\n",
    "def randomTrainingPair():\n",
    "    category = randomChoice(all_categories)\n",
    "    line = randomChoice(category_lines[category])\n",
    "    return category, line"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "对每一个时间点（也就是说在训练集中词的每个字母）网络的输入是 ``(类别, 当前字母, 隐藏层状态)`` , \n",
    "输出是 ``(下一个字母, 下一个隐藏层状态)`` . 对于每一个训练集, 我们需要的是类别、输入的字母集、输出/目标字母集. \n",
    "\n",
    "因为在每一步, 我们从当前的字母预测下一个字母, 这样的字母对是在原有行中连续字母的集合, \n",
    "例如, 对于 ``\"ABCD<EOS>\"`` 将会产生 (\"A\", \"B\"), (\"B\", \"C\"),\n",
    "(\"C\", \"D\"), (\"D\", \"EOS\"). \n",
    "\n",
    "![](https://i.imgur.com/JH58tXY.png)\n",
    "\n",
    "类别张量是一个大小为 ``<1 x n_categories>`` 的 [one-hot tensor](https://en.wikipedia.org/wiki/One-hot) 张量, \n",
    "在训练的每一个时间点把它喂给网络 —— 这是一个设计的选择, 它可以被当作为初始隐藏状或其他策略的一部分. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 类别的 one-hot 向量\n",
    "def categoryTensor(category):\n",
    "    li = all_categories.index(category)\n",
    "    tensor = torch.zeros(1, n_categories)\n",
    "    tensor[0][li] = 1\n",
    "    return tensor\n",
    "\n",
    "# 输入串从第一个字母到最后一个字母（不包括 EOS ）的 one-hot 矩阵\n",
    "def inputTensor(line):\n",
    "    tensor = torch.zeros(len(line), 1, n_letters)\n",
    "    for li in range(len(line)):\n",
    "        letter = line[li]\n",
    "        tensor[li][0][all_letters.find(letter)] = 1\n",
    "    return tensor\n",
    "\n",
    "# 目标的第二个字母到结尾（EOS）的 LongTensor \n",
    "def targetTensor(line):\n",
    "    letter_indexes = [all_letters.find(line[li]) for li in range(1, len(line))]\n",
    "    letter_indexes.append(n_letters - 1) # EOS\n",
    "    return torch.LongTensor(letter_indexes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了训练过程的便利, 添加一个 ``randomTrainingExample`` 函数, 获取随机的 (category, line) 对, \n",
    "并把他们转换成需要的 (category, input, target) 张量. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 从随机的（category, line）对中生成 category, input, and target 张量 \n",
    "def randomTrainingExample():\n",
    "    category, line = randomTrainingPair()\n",
    "    category_tensor = categoryTensor(category)\n",
    "    input_line_tensor = inputTensor(line)\n",
    "    target_line_tensor = targetTensor(line)\n",
    "    return category_tensor, input_line_tensor, target_line_tensor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 网络的训练\n",
    "\n",
    "\n",
    "与分类相比, 分类只用到了最后的输出, 而这里每个步都会产生一个预测, 所以我们需要计算每一步的损失. \n",
    "\n",
    "自动求导（autograd）的魔力就在于, 它允许将每一步的损失简单的加和, 并在最后调用 backward \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = nn.NLLLoss()\n",
    "\n",
    "learning_rate = 0.0005\n",
    "\n",
    "def train(category_tensor, input_line_tensor, target_line_tensor):\n",
    "    target_line_tensor.unsqueeze_(-1)\n",
    "    hidden = rnn.initHidden()\n",
    "\n",
    "    rnn.zero_grad()\n",
    "\n",
    "    loss = 0\n",
    "\n",
    "    for i in range(input_line_tensor.size(0)):\n",
    "        output, hidden = rnn(category_tensor, input_line_tensor[i], hidden)\n",
    "        l = criterion(output, target_line_tensor[i])\n",
    "        loss += l\n",
    "\n",
    "    loss.backward()\n",
    "\n",
    "    for p in rnn.parameters():\n",
    "        p.data.add_(p.grad.data, alpha=-learning_rate)\n",
    "\n",
    "    return output, loss.item() / input_line_tensor.size(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了跟踪训练花费了多长时间, 这里添加一个 ``timeSince(timestamp)`` 函数, 返回一个人们易读的字符串: \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import math\n",
    "\n",
    "def timeSince(since):\n",
    "    now = time.time()\n",
    "    s = now - since\n",
    "    m = math.floor(s / 60)\n",
    "    s -= m * 60\n",
    "    return '%dm %ds' % (m, s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "训练和往常一样, 不停的调用 train 并等待一会, 打印当前时间, 每隔 ``print_every`` \n",
    "个例子打印 loss, 将每 ``plot_every`` 个例子的平均损失保存在 ``all_losses`` 中以便后面画图. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0m 8s (5000 5%) 2.3679\n",
      "0m 16s (10000 10%) 2.3266\n",
      "0m 25s (15000 15%) 2.3716\n",
      "0m 34s (20000 20%) 1.8878\n",
      "0m 42s (25000 25%) 3.0048\n",
      "0m 50s (30000 30%) 2.5958\n",
      "0m 58s (35000 35%) 2.6039\n",
      "1m 7s (40000 40%) 3.3688\n",
      "1m 16s (45000 45%) 2.6047\n",
      "1m 25s (50000 50%) 2.8957\n",
      "1m 34s (55000 55%) 2.0515\n",
      "1m 42s (60000 60%) 2.3317\n",
      "1m 51s (65000 65%) 2.9110\n",
      "2m 0s (70000 70%) 2.5003\n",
      "2m 9s (75000 75%) 2.2145\n",
      "2m 19s (80000 80%) 2.4460\n",
      "2m 28s (85000 85%) 2.1489\n",
      "2m 36s (90000 90%) 2.7495\n",
      "2m 46s (95000 95%) 2.7960\n",
      "2m 55s (100000 100%) 2.0868\n"
     ]
    }
   ],
   "source": [
    "rnn = RNN(n_letters, 128, n_letters)\n",
    "\n",
    "n_iters = 100000\n",
    "print_every = 5000\n",
    "plot_every = 500\n",
    "all_losses = []\n",
    "total_loss = 0 # Reset every plot_every iters\n",
    "\n",
    "start = time.time()\n",
    "\n",
    "for iter in range(1, n_iters + 1):\n",
    "    output, loss = train(*randomTrainingExample())\n",
    "    total_loss += loss\n",
    "\n",
    "    if iter % print_every == 0:\n",
    "        print('%s (%d %d%%) %.4f' % (timeSince(start), iter, iter / n_iters * 100, loss))\n",
    "\n",
    "    if iter % plot_every == 0:\n",
    "        all_losses.append(total_loss / plot_every)\n",
    "        total_loss = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 绘制损失\n",
    "\n",
    "\n",
    "从 all\\_losses 中绘制历史损失, 以展现网络的学习过程\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAAypklEQVR4nO3deXiU1dn48e89S/adJBASIGEP+xLDJiCIvoALri11rwuuv1q7WtvqW9v39W2tba1aqVvdF1RQVNxZFJElQMIWlgCBACELIZB9Pb8/ZhInYSaZIGTCcH+uay5mznNm5n6eDPecOc95zhFjDEoppfyXxdcBKKWUOr000SullJ/TRK+UUn5OE71SSvk5TfRKKeXnbL4OwJ3Y2FiTnJzs6zCUUuqMsX79+mJjTJy7bV0y0ScnJ5ORkeHrMJRS6owhIvs8bdOuG6WU8nOa6JVSys9poldKKT+niV4ppfyc14leRKwislFEPnSzTUTknyKSIyKbRGSMy7aZIrLDue3+UxW4Ukop73SkRX8vkO1h2yxggPM2D3gaHF8OwFPO7UOAH4nIkJOOVimlVId5lehFJAm4CHjOQ5U5wMvGYTUQJSIJQDqQY4zZY4ypBd501lVKKdVJvG3R/wP4FdDoYXsikOfy+ICzzFP5CURknohkiEhGUVGRl2G19M8vd7Fi58k9Vyml/FW7iV5ELgYKjTHr26rmpsy0UX5ioTHPGGPSjDFpcXFuL+5q1/wVu/laE71SSrXgzZWxk4BLRWQ2EAREiMirxpjrXOocAHq5PE4CDgEBHspPiyC7ler6htP18kopdUZqt0VvjPmNMSbJGJMMzAWWtkryAIuBG5yjb8YDx4wx+cA6YICIpIhIgPP5i0/tLnwnyGahqtZT75JSSp2dTnquGxG5A8AYMx9YAswGcoBK4MfObfUicg/wKWAFXjDGbP2+QXuiLXqllDpRhxK9MWY5sNx5f75LuQHu9vCcJTi+CE67QLuVmjpN9Eop5cqvrowNtluortOuG6WUcuVXiT7IbqVaW/RKKdWC/yV67aNXSqkW/CzRa9eNUkq15l+J3qZdN0op1ZpfJfpAu1Vb9Eop1YpfJfogu0WHVyqlVCt+luj1ZKxSSrXmX4neZqWuwVDfoN03SinVxK8SfXCAY3eq6zXRK6VUE79K9EF2K4COvFFKKRf+lehtmuiVUqo1v0r0gXZn140OsVRKqWZ+lei160YppU7kl4m+RodYKqVUM/9K9DbtulFKqdb8K9Fr141SSp3ATxO9tuiVUqqJXyX6YG3RK6XUCfwq0Qc1Da/Uk7FKKdWs3cXBRSQI+AoIdNZ/xxjzUKs6vwSudXnNVCDOGFMiIrlAGdAA1Btj0k5d+C0FateNUkqdoN1ED9QA040x5SJiB1aKyMfGmNVNFYwxjwKPAojIJcB9xpgSl9eYZowpPpWBu9PcoteuG6WUatZuojfGGKDc+dDuvJk2nvIj4I3vH1rHBVgtiGiiV0opV1710YuIVUQygULgc2PMGg/1QoCZwLsuxQb4TETWi8i8Nt5jnohkiEhGUVGR1zvQ6jV0OUGllGrFq0RvjGkwxowCkoB0ERnmoeolwDetum0mGWPGALOAu0Vkiof3eMYYk2aMSYuLi/N+D1rRBcKVUqqlDo26McaUAstxtNrdmUurbhtjzCHnv4XAIiC9o0F2RJBdW/RKKeWq3UQvInEiEuW8HwzMALa7qRcJTAXedykLFZHwpvvAhcCWUxK5B47lBLVFr5RSTbwZdZMAvCQiVhxfDAuMMR+KyB0Axpj5znqXA58ZYypcntsdWCQiTe/1ujHmk1MWvRuBNou26JVSyoU3o242AaPdlM9v9fhF4MVWZXuAkd8rwg4KDtCuG6WUcuVXV8aCY5WpGj0Zq5RSzfwv0dstOgWCUkq58MNEr103Sinlyi8TfZUmeqWUauaHiV4vmFJKKVd+l+gDdQoEpZRqwe8SfZBdR90opZQrP0z0FmobGmlobGuCTaWUOnv4XaJvWk6wRodYKqUU4IeJXhcIV0qplvwu0QfaHLukLXqllHLwv0TvXE5QT8gqpZSD3yX6IFtTH70meqWUAj9M9IG6QLhSSrXgf4leW/RKKdWC3yX6ILuejFVKKVd+l+ibWvQ6vFIppRz8MNFri14ppVz5YaJ39tFri14ppQA/TPRNffS6ypRSSjm0m+hFJEhE1opIlohsFZE/uKlznogcE5FM5+1Bl20zRWSHiOSIyP2negda0xa9Ukq1ZPOiTg0w3RhTLiJ2YKWIfGyMWd2q3tfGmItdC0TECjwFXAAcANaJyGJjzLZTEbw7zVfG6vBKpZQCvGjRG4dy50O78+btHMDpQI4xZo8xphZ4E5hzUpF6KcCqF0wppZQrr/roRcQqIplAIfC5MWaNm2oTnN07H4vIUGdZIpDnUueAs8zde8wTkQwRySgqKvJ+D1qxWIQAm0Vb9Eop5eRVojfGNBhjRgFJQLqIDGtVZQPQxxgzEngCeM9ZLu5ezsN7PGOMSTPGpMXFxXkTlkeBNosOr1RKKacOjboxxpQCy4GZrcqPN3XvGGOWAHYRicXRgu/lUjUJOPQ94vVKoM2qLXqllHLyZtRNnIhEOe8HAzOA7a3q9BARcd5Pd77uEWAdMEBEUkQkAJgLLD6le+BGkN2iffRKKeXkzaibBOAl5wgaC7DAGPOhiNwBYIyZD1wF3Cki9UAVMNcYY4B6EbkH+BSwAi8YY7aejh1xFah99Eop1azdRG+M2QSMdlM+3+X+k8CTHp6/BFjyPWLssECbVcfRK6WUk99dGQuOsfR6MlYppRz8MtEHaYteKaWa+WWi1xa9Ukp9xz8Tvc2i89ErpZSTnyZ6q7bolVLKyS8TfZBdh1cqpVQTv0z0emWsUkp9x08TvV4Zq5RSTfwy0QfZtUWvlFJN/DLRB9osNDQa6hs02SullH8m+uZ1YzXRK6WUfyb65nVjtZ9eKaX8MtEH6bqxSinVzC8TfVOLXkfeKKWU3yZ6bdErpVQTv0z0QXZnH70meqWU8s9E39Si164bpZTy10SvJ2OVUqqZfyZ6HV6plFLN/DLR6/BKpZT6TruJXkSCRGStiGSJyFYR+YObOteKyCbnbZWIjHTZlisim0UkU0QyTvUOuKPDK5VS6js2L+rUANONMeUiYgdWisjHxpjVLnX2AlONMUdFZBbwDDDOZfs0Y0zxqQu7bTq8UimlvtNuojfGGKDc+dDuvJlWdVa5PFwNJJ2qAE9Gcx+9JnqllPKuj15ErCKSCRQCnxtj1rRR/RbgY5fHBvhMRNaLyLw23mOeiGSISEZRUZE3YXnUPKmZdt0opZR3id4Y02CMGYWjpZ4uIsPc1RORaTgS/a9diicZY8YAs4C7RWSKh/d4xhiTZoxJi4uL68g+nEC7bpRS6jsdGnVjjCkFlgMzW28TkRHAc8AcY8wRl+cccv5bCCwC0k8+XO+ICAE2iy4QrpRSeDfqJk5Eopz3g4EZwPZWdXoDC4HrjTE7XcpDRSS86T5wIbDllEXfhiCbhZo6bdErpZQ3o24SgJdExIrji2GBMeZDEbkDwBgzH3gQ6Ab8S0QA6o0xaUB3YJGzzAa8boz55NTvxomCA6yU19R3xlsppVSX5s2om03AaDfl813u3wrc6qbOHmBk6/LO0DsmhH1HKnzx1kop1aX45ZWxAP3jw9hdpIleKaX8NtH3iwujpKKWkopaX4eilFI+5deJHmBPUXk7NZVSyr/5faLfrYleKXWW89tEnxgdTIDNQk6hJnql1NnNbxO91SL0jQ3VE7JKqbOe3yZ6gH7xYdp1o5Q66/l3oo8LI6+kUic3U0qd1fw60Q/qHk6jgR2Hy3wdilJK+YxfJ/q05GgA1uWW+DgSpZTyHb9O9N0jgkjuFsLqPZrolVJnL79O9ADpKTGsyy2hsdG0X1kppfyQ3yf6cSndOFZVx85C7adXSp2d/D7Rp6fEALB2r3bfKKXOTn6f6HvFhJAUHcxLq3IpPF7t63CUUqrT+X2iB/jLVSPIP1bN1f/+luPVdb4ORymlOtVZkegn9otl/nVj2Xekki+2Ffg6HKWU6lRnRaIHOLd/LLFhgSzdXujrUJRSqlOdNYneYhGmDYrjq51F1DfoouFKqbPHWZPoAaYPjud4dT3r9x31dShKKdVp2k30IhIkImtFJEtEtorIH9zUERH5p4jkiMgmERnjsm2miOxwbrv/VO9AR5w7IBa7VfgiW/vplVJnD29a9DXAdGPMSGAUMFNExreqMwsY4LzNA54GEBEr8JRz+xDgRyIy5NSE3nHhQXamD47npVX7WL3niK/CUEqpTtVuojcOTZO625231vMJzAFedtZdDUSJSAKQDuQYY/YYY2qBN511febPV46gd7cQbns5Q+eqV0qdFbzqoxcRq4hkAoXA58aYNa2qJAJ5Lo8POMs8lbt7j3kikiEiGUVFRV6G33FRIQG8+ONzsFst3Pnqeipr60/beymlVFfgVaI3xjQYY0YBSUC6iAxrVUXcPa2Ncnfv8YwxJs0YkxYXF+dNWCctKTqEx+eOYldhOX/8cNtpfS+llPK1Do26McaUAsuBma02HQB6uTxOAg61Ue5zkwfEccP4Pryz/gBFZTW+DkcppU4bb0bdxIlIlPN+MDAD2N6q2mLgBufom/HAMWNMPrAOGCAiKSISAMx11u0SbpiYTF2DYUFGXvuVlVLqDOVNiz4BWCYim3Ak7s+NMR+KyB0icoezzhJgD5ADPAvcBWCMqQfuAT4FsoEFxpitp3gfTlq/uDAm9O3G62v206Dz1Sul/JStvQrGmE3AaDfl813uG+BuD89fguOLoEu6dnxv7nl9I6t2FzN5wOk9N6CUUr5wVl0Z6875g7sTYLXw9a5iX4eilFKnxVmf6IMDrIzuHcWq3cUYY1i2vZDqugZfh6WUUqfMWZ/owTGN8dZDx3k/8xA/fnEdTy/f7euQlFLqlNFED0zq3w1j4HfvbQHg1dX7tFWvlPIbmuiBEUlRhARYKa+p57+GdudIRS2Ls7rEcH+llPreNNEDATYLE/p2o1toAH//4SgG9wjnhZV7cQwmUkqpM5smeqdHrhzOu3dOJCTAxs2TUth+uIxvd+sMl0qpM58meqf48CCSY0MBuHRUT7qFBvDCN3tZv+8o72086OPolFLq5LV7wdTZKMhu5drxfXhi6S6Wbi+k0cC4vjEkRAb7OjSllOowbdF7cN343nQLDWD64O4AfLQp38cRKaXUydFE70F8eBBrH5jBczemMTwxkg800SulzlCa6NtgsTim0794RAJZeaXsP1Lp44iUUqrjNNF74aIRCQDM/0qvmFVKnXk00XshKTqE2yan8Pqa/by+Zr+vw1FKqQ7RRO+l+2elMnVgHA8t3kJucYWvw1FKKa9poveS1SI8etUI7FYL//dx6wW2lFKq69Jx9B0QHxHEXef146+f7eQXb2cRYLMwLiWGC4f0IDjA6uvwlFLKLU30HXTr5L58kJXPF9kFNDQaXl+zn/MHx/P8Tef4OjSllHJLE30HBdmtfHrfFAAaGg3/WpbDY5/vZOn2Aob2jORIeS2x4QHEhwf5OFKllHLQRP89WC3C7VP7sSjzID95I5OK2nqMARH4/L6p9I8P83WISinV/slYEeklIstEJFtEtorIvW7q/FJEMp23LSLSICIxzm25IrLZuS3jdOyELwXYLDxy+XD6dAvh/00fwF+uGoExsGq3rkGrlOoapL0510UkAUgwxmwQkXBgPXCZMWabh/qXAPcZY6Y7H+cCacYYrzNfWlqaycg4M78TjDFMeGQp56TE8MSPRvs6HKXUWUJE1htj0txta7dFb4zJN8ZscN4vA7KBxDae8iPgjZMJ1B+ICGnJ0azbW6ILlyiluoQOjaMXkWRgNLDGw/YQYCbwrkuxAT4TkfUiMq+N154nIhkiklFUVNSRsLqc9JQYDh+v5mBpla9DUUop7xO9iIThSOA/NcYc91DtEuAbY0yJS9kkY8wYYBZwt4hMcfdEY8wzxpg0Y0xaXFyct2F1SWl9YgDIyD3q40iUUsrLRC8idhxJ/jVjzMI2qs6lVbeNMeaQ899CYBGQfnKhnjkG9QgnPNDGmr26FKFSyve8GXUjwPNAtjHmb23UiwSmAu+7lIU6T+AiIqHAhcCW7xt0V2e1CNNT41m44SC5xRWs31fCulzHj5w/friNm19cp/33SqlO4804+knA9cBmEcl0lj0A9AYwxsx3ll0OfGaMcZ3xqzuwyPFdgQ143RjzySmIu8t7YHYqS7MLueGFtRw4WondauGhS4by/Mq9AHyRXcgFQ7o3129sNHy27TAvf7uP26b0ZdqgeF+FrpTyM+0Or/SFM3l4pas31u7nNws3M3NoDzL2lVBcXktCZBB2q4WwQBuPzx2FAXrHhPCTNzby2bYCAMb0jmLhXZN8G7xS6ozS1vBKvTL2NPpRem/SU2LoGxvKypxi7nhlPQ9dMoTymgZ+8XYWF/z9KwCiQ+wcrazjt7NTqW80/PmT7eQUltE/PtzHe6CU8gea6E+zfnGOaRAmD4gj86ELsVst1Dc0UlpZS1RIAOXVdSzZfJhrx/dmzqhEistreOyzHSzIOMADs1N9HL1Syh9oou9Edqvj3LfNauHWyX2by2+alNJ8PzYskPNT43l3/QHumzFQpz9WSn1vuvBIF3TzpBSOVNTyn1V7fR2KUsoPaIu+CxrXtxvnD47n6WW7qapt4MvsQo5X19GnWwhzRiXyg7Revg5RKXUG0RZ9F/XrWYOpqK3niaU5RAbbSesTTf6xan71zia2HDzm6/CUUmcQbdF3UQO7h/ParePpFhbAwO6O0TeHj1Uz/pEvWb3nCMMSI5vrbj10jN8u2sJT144hMSq43deua2jkwr9/xU/O78/lo5NO2z4opboGbdF3YRP6dWtO8gA9IoPoHRPC2r3fTSV0vLqOu17bQGZeKcu2F3r1ujsOl7G3uILVu0var6yUOuNpoj/DpKfEsC63hH1HKrjpP2uZ9Y+vOXC0ipAAKxv3l3r1Gpl5jnp7j1S0XVEp5Rc00Z9h0pNjOFpZx20vZ7BubwlDe0bw+NxRTOwXy8b97mfLzM4/zidb8psfbzpQCkBusSZ6pc4G2kd/hklPcUyBvLOgnN9dlNo8Hn/fkUq+yC7gaEUt0aEBzfWNMfxsQRa7C8vJfCiOkAAbWXmOk7mFZTVU1NQTGqgfA6X8mbbozzB9uoXQIyKIvrGh3DAhubl8dO8oANbllrBo4wGOVdUBsDKnmOz849Q2NPLt7iNU1NSzq7CMAc6Fy3O1+0Ypv6dNuTOMiPDcjWmEB9kIsH33PT0yKQqLwM/fzqKsup6+saH86bJh/GvZbuLCA6moqWf5jiLCAm00GrhsdCKPfrqD3OJKhvaMdPter6/ZT3x4IDNcZtlUSp15tEV/BhqWGEmfbqEtykIDbQzqEUFZdT23nJtCaVUd1zy3hm/3HOHHk5KZ2K8by3cWsmq3YzGUS0f2BDy36KvrGnj4w6386t1NVNTUn94dUkqdVtqi9yMPzxlKaWUdFwzpzp3n9SNzfylWqzCpXyxvBeXxRXYh/1y6iykD4+gVE0J8eCB7PZyQXZdbQnVdI9V1tbz0bS53nde/k/dGKXWqaKL3I+ckxzTfjw1r2eUybVAcVoswtk80868bA0BybKjHkTcrdhQRYLWQlhzNM1/t4YYJyYTpSVulzkjadXOWSIoO4bP7pvDyzemEBDgSdkq3ULYcOsa0vy7n7tc2sLuovLn+8p1FjOsbw13n9ae0so6M3BJKK2v52YJMFmcdor6h0Ve7opTqIG2inUWa5sZvkp4Sw+KsQyRFB7N8RyGfbD3M1WOTmNCvGzmF5cw9pxcjejlO1G49dJyishoWbjjIwg0Heb5XFC/fnE5ksN0Xu6KU6gBN9GexK8cmccWYRESEI+U1PLVsN6+u3seb6/IQgWmD44kIspPcLYQtB48RHRpAeKCNP8wZyq/f3cT1z6/hlVvGNSf7kopaistrWkzb4GpXQRndI4OICNIvB6U6kyb6s5xz4Xa6hQXy4CVDuGtaPw46p1Ro+gUwNDGSTQdKCQ2wMap3FFeMSSIy2M4dr67nhufX8PIt4wi2W7n++TUcOFpFxu9mNC+y0qSytp5Ln/yGy0b35JErRnT6fip1Nmu3j15EeonIMhHJFpGtInKvmzrnicgxEcl03h502TZTRHaISI6I3H+qd0CdWrFhgYzsFcUAl1b5sJ6R5JVUsaOgjNG9owE4P7U7T187lm35x7ny6VX8/O0sth46zrGqOtbtLeHFb/ZyyRMraWh0LD6/clcxVXUNLNl8mLpW/fv/89E2Xl+zv/N2UqmzjDcnY+uBnxtjUoHxwN0iMsRNva+NMaOct4cBRMQKPAXMAoYAP/LwXNWFDUuMAMAYGOO8AhdgxpDuPH/jOdQ1NPJB1iEuHdmTAJuFT7ce5ukVu9l88Bjr9znm31nqnFnzWFUd3+QUN7/G3uIKnv16Ly9/m9tp+6PU2abdRG+MyTfGbHDeLwOygUQvXz8dyDHG7DHG1AJvAnNONljlG65Xzo7uFd1i25SBcXzxs6m8dHM6f7lqBBP7deO1NfspOF4DwJLN+TQ2GpZuL2RGanfCA20s2fzdBGuvrd4HwI6CMsqqHdM2HCytYt7LGeQUlrV4L2MMy7YXUlXbcFr2Uyl/1aHhlSKSDIwG1rjZPEFEskTkYxEZ6ixLBPJc6hzAw5eEiMwTkQwRySgqKupIWOo0iwkNIDEqmH5xoUSGnHgi1W61MHVgHEF2K+endqe+0ZAQGcT5g+P5ZMthNh88RmFZDbOH92DGkO58vOUwn2w5TF5JJW+vP0BiVDDG0DzN8vzlu/lsWwF3v7aR6rrvkvryHUX8+MV1vLvhQGftulJ+wetELyJhwLvAT40xx1tt3gD0McaMBJ4A3mt6mpuXMu5e3xjzjDEmzRiTFhcX521YqpPcd8FAfjpjYLv1ZqTGY7cK147rzcUjEzh8vJp5r2RgtwrnDYrntsl9iQhynMid/JdlHKuq4+E5Q7EIZOw7ytGKWt5en8fwxEh2FJTxfx9vBxyt+SeX5QCwLb/1x08p1RavRt2IiB1Hkn/NGLOw9XbXxG+MWSIi/xKRWBwteNeVrJOAQ98vZOULV431bsnBhMhgvvjZVJKiQ6iorSc8yEZYoI1HrxpJTGgAMaEBrPjleXy1q4jC4zXEhgVyfmp3BvWIYMO+o1hFqK5r5LEfjGT+it28s/4AD8xOZeP+o6zfdxSrRdjuJtFn5pXSNy7U7dDNxkZD1oFSSqvqGNYzkrjwwO91LKpqG9hfUsmgHu6HkSrV1bSb6MUx/u55INsY8zcPdXoABcYYIyLpOH4pHAFKgQEikgIcBOYC15yi2FUX1TThWkSQnZW/mk5ooBWby3BLm9XC9MEtZ8RM6xPNm+v2s2p3MTNSuzOwezizhiWwcMNB1uWW8NzXe4gNC2BGancWZx2isdFgsTh+MD739R7+9FE2Q3tG8Ppt40+4iGvhxoP84u0sACYPiOWVW8a12N7QaDhUWkV9oyG5W0jzkFNPnvt6D08szSHj9zO65DUBh49VU9/YSFJ0iK9DUV2EN103k4Drgekuwydni8gdInKHs85VwBYRyQL+Ccw1DvXAPcCnOE7iLjDGbD0N+6G6qMgQe4sk78m4vjHUNRimD+7O43NHAXBu/1gCbRaeX7mXZTuKuG58H0b1iqKytoG8o5Ucr67jofe38KePshnfN4adBWX88N/f8tzXe8jOP44xjl7CL7YV0CMiiOvG92ZlTjH5x6qa3/doRS2XPfUNk/+yjGl/Xc5P3syksvbE2Tr/7+PtzPzHVxhj2HTwGLUNjWw+cOzUHKRT7OdvZ3LXaxt8HYbqQtpt0RtjVuK+r921zpPAkx62LQGWnFR06qwxe1gCC24PYmyfaKzOlnpwgJWJ/bqxdHshdqtwzbje5JdWA7B6zxEe/2IX+ceruWliMr+7KJXlO4r434+z+dNH2QD0jQ3lrdsnsDKnmEtG9uSWc/vy6ur9vJ95iDum9qPweDU3vLCWPcUV/O6iVEor6/jX8hyKyqp5c96E5tjeXLuf+St2A7C/pJIdhx2jgTLzSpnUP7YzD1O76hoaWb/vKI2NjvutL1xTZye9MlZ1CRaLNC+T6Or81O4s21HExSN6Eh8eRHigHYvAIx9v51hVHW/cNp7xfbsBjnH9M4Z051BpFV9uL+T3723h/72xgfKaes4bFEdKbCijekWxcMMB+seF8fv3t1BaWcdzN6QxZaBjAEBUiJ0/fZRNVl4pI3tFkVdSye/f38LA7mHsLCjnm5wj7C+pBCDLuci6OwXHq1mXW8K0QfGdulTj9vwyquscF6TlFle0uPBNnb306151aTOH9SA9JYY7z+sHOFr5ybGhlFbWccmIns1J3lXPqGCuH9+H2cN7sHpPCXarNLe8rxyTyM6Ccm59OQMB3rlzQnOSB/jhOb0ICbDyinN8//wVuxGEF246hyC7hQUZjtHCMaEBZOaVNncPNdl/pJIHFm1m8p+Xcc/rG5n+2HK+2nlyw4ULjlfzfubBE96jLRtcFojffrisjZrqbKKJXnVpsWGBLLh9QouJ0oYkRGC1CPdd0PZwz3vPH4iIY57+prn0f3hOb/7xw1G8fus4PvvZ1BOWUQwPsnP56EQ+yDrE2r0lvJ1xgCvHJpEUHcKwnpFkOlvxV45JpLCshh0FZaxyXum7dHsB0x5bzjsZB7gqLYlnb0gjLNDGz9/Ooqa+4xd5/XvFHu59M5NXOzA9xPp9R4kNC8BqkeYuJqW060adce67YCCXj04kJTa0zXqDeoTz5ytH0D/+u+mZA2wWLhvd9oXdN05MZkFGHj/497dYBO6Y2heAUb2iyNh3lPBAG7OGJ/Ds13u58l+rqKht4C9XjeDJpTn0iwvl5ZvH0SMyCIAgu4Xrn1/L4sxDXJ3Wq623PcG63BIAHv5gK3aLMGtYgtsL1lxt2H+U9JQYdhaUa4teNdNEr844/eLCTphb35MfdDC5AgzsHs4nP53CqpxiokICmoeLjuwVBcDghHCGJEQQaLMQaHd0Jf3qnU0AvHJLenOSB8fIocE9wnnu671cNTap3aGbTcpr6tl66Bg3TujDmr0l3L9wMw8t3sodU/tx53n9CLJbT3hOYVk1B45WcdPEZESETQdKm7cZY7x+b+V/tOtGKTf6xYVx/YRkLnEuog6OFj3A4B4RBNmtLLh9Akt+Mpl/Xz+WyGA7M1K7M3lAy6u6RYRbJ/dlR0EZX2QXun2vN9bu55YX17Xoatm4/yiNxnEyeslPJrPorolcMKQ7j3+5i/veynT7OutzHf3zY/pEM7h7OHklVZTX1LPjcBkT/28pn2zJd/u8fUcq+HqX5/MItfWNXX5FsZzCct7OyGu/4llKE71SXkqKDub2qX2bfyWM7BVFj8ggkqJDWPHL83jauRZva3NG9SQlNpRHP93ePG1zZl4pn2w5TGFZNX/8cBtfbi/kon9+zQrnidt1e0uwiCNpWyzC6N7RPHnNGH52wUA+3nK4+byAq9V7jhASYGV4YmTzVbufbT3MT9/KJP9YNX/8MLvF3EHGGP704TbOf2wF1z+/lp0F7rt6fvDvb3lg0eaTP3Cd4PmVe/nlO5vIc46IUi1polfKSyLCb2alMjwp8oRtUSEBHses260WfnHhIHYWlPPO+jxyCsu57rk13PHqeq55dg219Y0svGsisWGBvOKcrnltbglDe0aesCD7vCl9SYoO5g8fbDuhlb1mbwlj+0Rjt1pIT4khKTqYny3IIjv/OLdP6cvB0ir+801uc/3PtxXw3Mq9XOKcXvqlVbm0truonMy8Uj7alN/iS8KXjDF8uOkQFTXfXdjWtN7x4iydYcUdTfRKdYLZw3swqlcUv353M5c/9Q1BdgsXDU9wJP3xfRjTO5pLRiawYmcRucUVbNxfSlpy9AmvE2S38vuLh7CjoKx5kjdwXOG7/XBZ83DTqJAAvvjZVH53USoPXjyE38xO5fzB8Ty9PIfqugbqGxp59NMd9I0L5dGrRnDZqJ4s3HCQY5V1Ld7vky2HAaiobWDV7hN/RXijuq6BC/++goUdmHU0r6SSorIat9sy9h3lntc38qLLF9OepkSfqYneHT0Zq1QnEBFe+nE6r67Zx2fbCvj9RamM6R3NFWMSm8f4XzyiJ89+vZfrnl9DfaNh7jm93b7Wfw3twRWjE3liaQ6NjQYDJEYFAzDO5aKzILuVWyf3bX58y7kpfLm9kE+2HKa6roFdheU8fe0YbFYLN01MYUHGAd5Yt587pvZjXW4J0SEBfLLlMMMSI9hXXMmnWwqICgkg2G4lNSHC474+uXQXn2w9zILbJxASYOOL7AJ2FpTz0aZ8rhiTxEeb8hnSM8LtqKm8kkpuezmD7YfLCLBauGZcb353UWqLaTQ+2uQ41/D5tgLuntaf0spaistrSYkNZUdBGdn5x9uMz1Vjo6HBGL+/glgTvVKdJDLEzt3T+nP3tP7NZeenfje524ikSHrHhLC/pJIfT0puc3bM/54zlHX7Svjn0u9a9UF2CyOSojw+Z3zfbvTpFsLzK/eSd7SStD7RzBzWA4AhPSOYMjCOf6/YTXpKDNc8uxpBqG1o5DezBrPl0HHeyzzIgvV5hAXa+OCec0l2SdSr9xyhqq6Byf1jeXHVPorLa/jjh9k8csVwFm44CDi6owqOV3PPGxu4emwSf7lq5AkxPvv1HvYUV/DgxUPYcugYL67KZVL/WC4Y4jhOjY2Gj7fkY7UImXmlFJZVN/fL33VePx5Y5Bid9J+bzmn3iuSDpVVc99waRvWK4u8/HEVhWTX1DYaezi9Nf+LfX2NKnUFEhLnpvUiMCm537v+IIDuf3DuFzAcv4N07JxIbFsC5/eMIsHn+L22xCD9I68Xmg8eoqKnnkSuGtxhy+csLB3G0so5rnl1NaKCNqYPiCLJbmD08gUtH9qSmvpFLRvTEahHueHU9VbWOLqCH3t/C3GdWc/vL63lzXR7F5TWM6hXFG2v389hnO1ixs4jkbiGUVdfz+Je7MAay8x0nfv/66Q7ez3R8EVTW1rNow0EuGp7Azeem8Mc5wwDIdpmWesP+oxQcr+E25y+VL7ML2V1YAUB6Sgx/vXokGbklXP/8GgqOV7fYf2NM83rFh0qr+OG/v2VvcQUrnSe2f/H2Ji576pvmlc7a837mQeY89Q019Q1U1zWwbEchhWXV7T+xDY2N3l8F3RHaoleqC7nrvP7cMaVf8xTMbWlqsY7tE8DXv5qOcb+mTwtXj03iyaU53HlevxPmwRmeFMlFIxL4aFM+D88ZxtVjk6iuayQ4wEqvmBBW3T+dnlHBLN9RyE3/Wcejn+4gJtTOS9/u45pxvVm04SAPvr+FyGA7r946jvveyuQJ5y+Oh+cM44YX1vLmWsdVvjsLyjheXcfTK3ZjtQhDEiLYsP8oZTX1XDuud/P+9Y4JaTHsdHHWIQJsFu6e1o8PNx3i820FDIgPI8BmISk6hD7dQrFbLfx8QRYz//EVf75yBBP7x/LwB1v5MruQ8pp6fjVzMG9n5HGsso4rxyTx7oYDFB6vZuM+x/s/9tlO/vvSobSlsraeP32UTVFZDd/kFJO5v7T519V9MwZy74wBPLUsh6ToYOaM8nyBXnlNPZW19cSHB/HSqlxW7CziX9eOcXudxPehiV6pLsabJN9acIB3iSE+Iog1vz2fcA/dGn+cM4wZqfFcNioREWnxuk1dGucNiuf68X34z6q92CzCRSMS+N/LhxMbFsg/v9zFRSMSCAu08cz1Y/l4y2EOHq1iysA4esUEk1dSRWJUMAdLq3hv48Hm4aY3vrCW8pp6BnYPY2yf705CD+oRzvbDjhb93uIK3li7n8tGJRIeZOei4Qk8t3IveSWV9I0NbZ71dPbwBAb1COcnb2xk3ivriQ0L5GhlLXNG9qSwrIY/frgNm0V48cfpWC3CuxsO8NHmfMpq6kmMCublb3O5bHRi83UT7ry4KpeishoCbBY+yMpn9Z4jnJMcTUSQnaeW55CaEM6jn+4gMSqYS0f29Hix2u2vZLAu9ygXD09g4caDXDCke/N+nEradaPUWSYiyO4x8cSEBnD56Pav4L1/1mASo4KJCLLzsLP1e/uUvlw9Nolbz00BHF1Rs4cncNsURzfLuBTHiKB7pjvOUbzy7T5E4G8/GInVKkwbHM8/fji6xXun9ghnb3EF1XUN/M9H2QRYLfzyvwY53m9qP0IDrOwqLD/hSul+cWEsvGsi86b0dUxSd3M6f/vhKF66OZ3fXZTKv64dw7kDYhniPGn71jrHxVaPzx1FQmQw9765kbLqOnIKy6itbzmM9dOth3lqaQ7TB8dz6cievJd5kPxj1fx4UgoPXTKUxkbTvB7AwdIqNnlYt2BVTjHf5BwhKSqYhRsPMn1wPE9eM/q0nBjWFr1SqsNCA20sumsSdQ2NdAsLbC579OoTT7A2uWFCHyKDHZPG/f69LewqLGdwj3DmjEr02L0xqEcEjcZx9fAX2QX8euZg4iMcU0zEhAZw3wUD+cMH2+gXd+IInkCblQdmp/LA7NTmMqtFWoxEigyxkxgVzPbDZQTbrc0nZuc+8y3p//MlVXUNxIUHctvkFG49ty9PLM3h71/sZHhiJH+8bBg7C8p4Z/0BokPsnJ8aT6DNypVjkngrI49fXDiQx7/cxUeb85unzwB44stdZB0o5cDRKhIig1hy72R2HC4jNSGizXMs34cmeqXUSeno2rsjkqKaRwX1jw9j++GyFt007jSNPPrrpzuIDrFz08TkFtuvG9+Hw8equbSNfvD2pCaEc7C0imGJEdicF5s9dMlQVu0uZlL/WL7ILuR/l2znvY2H2JZ/nCvGJPLIFcMJtFmJCwskITKIy0cnEmhzdHP9cuYgkmNDuXVyChn7jvLRpnwuHdmTpOhgdhdV8LcvdhJks1JV18AjVwwnyG5t8UVwOmiiV0p1usE9wtl+uMztRWGukruFEGCzUFHbwC2T+55wLsJutfAblxb7yUhNiOCL7MIWQ1NvnJjMjc4vlevH9+H5lXv5nyXZXDQ8gUevGtncjx5gs7D05+e1aInHhgU2r59w8Yie/OLtLC5+YiWhAVYigu30iAji0/umUHi82uvJ+b4vTfRKqU43LDGS9zIPkdbnxFXFXNmsFgbEh5FTWM4NE/qcllia+ulHuJnaAr6bmO6SkT2JCws84WR5WyfCLxvVk4ggG43GsDjrEJ9uLeCZ68cSEWTv1IXlpSOr13SWtLQ0k5GR4eswlFKnSVVtA1sPHSMtue1ED46Tn2XV9Vw1Num0xFJd18DTy3dz+9S+hASc3rZvdV3DKR862URE1htj0txuay/Ri0gv4GWgB9AIPGOMebxVnWuBXzsflgN3GmOynNtygTKgAaj3FIgrTfRKKdUxbSV6b76+6oGfG2M2iEg4sF5EPjfGbHOpsxeYaow5KiKzgGeAcS7bpxljTm5GJKWUUt9Lu4neGJMP5Dvvl4lINpAIbHOps8rlKauB0/MbSymlVId1aNCmiCQDo4E1bVS7BfjY5bEBPhOR9SIyr43XniciGSKSUVTkebUbpZRSHeP1mQcRCQPeBX5qjDnuoc40HIn+XJfiScaYQyISD3wuItuNMV+1fq4x5hkcXT6kpaV1vTPESil1hvKqRS8idhxJ/jVjzEIPdUYAzwFzjDFHmsqNMYec/xYCi4D07xu0Ukop77Wb6MUx8cTzQLYx5m8e6vQGFgLXG2N2upSHOk/gIiKhwIXAllMRuFJKKe9403UzCbge2Cwimc6yB4DeAMaY+cCDQDfgX84JiZqGUXYHFjnLbMDrxphPTuUOKKWUaps3o25WAm1OZWeMuRW41U35HsDzLEdKKaVOuy55ZayIFAH7TvLpsUBXHLOvcXVcV41N4+oYjavjTia2PsaYOHcbumSi/z5EJMObq287m8bVcV01No2rYzSujjvVsenCI0op5ec00SullJ/zx0T/jK8D8EDj6riuGpvG1TEaV8ed0tj8ro9eKaVUS/7YoldKKeVCE71SSvk5v0n0IjJTRHaISI6I3O/DOHqJyDIRyRaRrSJyr7P8v0XkoIhkOm+zfRRfrohsdsaQ4SyLEZHPRWSX89+2F/I89TENcjkumSJyXER+6otjJiIviEihiGxxKfN4fETkN87P3A4R+S8fxPaoiGwXkU0iskhEopzlySJS5XLs5ndyXB7/dp11zDzE9ZZLTLlNV/t38vHylCNO3+fMGHPG3wArsBvoCwQAWcAQH8WSAIxx3g8HdgJDgP8GftEFjlUuENuq7C/A/c779wN/9vHf8jDQxxfHDJgCjAG2tHd8nH/XLCAQSHF+Bq2dHNuFgM15/88usSW71vPBMXP7t+vMY+YurlbbHwMe9MHx8pQjTtvnzF9a9OlAjjFmjzGmFngTmOOLQIwx+caYDc77ZUDTQi1d2RzgJef9l4DLfBcK5wO7jTEne2X092IcU2iXtCr2dHzmAG8aY2qMMXuBHE7j7KzuYjPGfGaMqXc+9MmiPx6OmSeddszaiss5WeMPgDdOx3u3pY0ccdo+Z/6S6BOBPJfHB+gCyVVOXKjlHudP7Bc6u3vEhbuFYLobx0piOP+N91FsAHNp+Z+vKxwzT8enq33ubqbloj8pIrJRRFaIyGQfxOPub9dVjtlkoMAYs8ulrNOPV6sccdo+Z/6S6N1NuubTcaNy4kItTwP9gFE4lmZ8zEehTTLGjAFmAXeLyBQfxXECEQkALgXedhZ1lWPmSZf53InIb3Gs7/yasygf6G2MGQ38DHhdRCI6MSRPf7uucsx+RMsGRacfLzc5wmNVN2UdOmb+kugPAL1cHicBh3wUi9uFWowxBcaYBmNMI/AsPlqAxbhfCKZARBKcsScAhb6IDceXzwZjTIEzxi5xzPB8fLrE505EbgQuBq41zk5d58/8I87763H06w7srJja+Nv5/JiJiA24Anirqayzj5e7HMFp/Jz5S6JfBwwQkRRnq3AusNgXgTj7/k5YqKXpD+h0OT5YgEU8LwSzGLjRWe1G4P3Ojs2pRSurKxwzJ0/HZzEwV0QCRSQFGACs7czARGQm8GvgUmNMpUt5nIhYnff7OmPb04lxefrb+fyYATOA7caYA00FnXm8POUITufnrDPOMnfSmezZOM5e7wZ+68M4zsXxs2oTkOm8zQZeATY7yxcDCT6IrS+Os/dZwNam44Rj0ZgvgV3Of2N8EFsIcASIdCnr9GOG44smH6jD0ZK6pa3jA/zW+ZnbAczyQWw5OPpvmz5r8511r3T+jbOADcAlnRyXx79dZx0zd3E5y18E7mhVtzOPl6cccdo+ZzoFglJK+Tl/6bpRSinlgSZ6pZTyc5rolVLKz2miV0opP6eJXiml/JwmeqWU8nOa6JVSys/9f/lIRD9XX1fFAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(all_losses)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 网络采样\n",
    "\n",
    "\n",
    "为了采样, 我们给网络一个字母并问下一个字母是什么, 重复这个过程直到 EOS 标记. \n",
    "\n",
    "-  创建输入类别、起始字母和隐藏层状态的张量\n",
    "-  创建一个带有起始字母的 ``output_name`` 串\n",
    "-  直到最大的输出长度, \n",
    "\n",
    "   -  当前字母喂给网络\n",
    "   -  从最高的输出获取下一个字母和下一个隐藏层状态\n",
    "   -  如果输出字母是 EOS, 算法结束\n",
    "   -  如果输出是常规字母, 将其加入到 ``output_name`` 并继续\n",
    "\n",
    "-  返回最终的名字\n",
    "\n",
    "Note:\n",
    "\n",
    "与给定起始字母不同的是, 有其他的策略是在训练的时候包含一个“串起始”标记, 让网络选择属于自己的起始字母. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rovakov\n",
      "Uantov\n",
      "Shantan\n",
      "Gangent\n",
      "Eren\n",
      "Roure\n",
      "Santaraz\n",
      "Pares\n",
      "Allan\n",
      "Chan\n",
      "Han\n",
      "Iun\n"
     ]
    }
   ],
   "source": [
    "max_length = 20\n",
    "\n",
    "# 从类别和起始字母采样\n",
    "def sample(category, start_letter='A'):\n",
    "    with torch.no_grad():  # no need to track history in sampling\n",
    "        category_tensor = categoryTensor(category)\n",
    "        input = inputTensor(start_letter)\n",
    "        hidden = rnn.initHidden()\n",
    "\n",
    "        output_name = start_letter\n",
    "\n",
    "        for i in range(max_length):\n",
    "            output, hidden = rnn(category_tensor, input[0], hidden)\n",
    "            topv, topi = output.topk(1)\n",
    "            topi = topi[0][0]\n",
    "            if topi == n_letters - 1:\n",
    "                break\n",
    "            else:\n",
    "                letter = all_letters[topi]\n",
    "                output_name += letter\n",
    "            input = inputTensor(letter)\n",
    "\n",
    "        return output_name\n",
    "    \n",
    "# 给定一个类别和多个起始字母 获取个采样结果\n",
    "def samples(category, start_letters='ABC'):\n",
    "    for start_letter in start_letters:\n",
    "        print(sample(category, start_letter))\n",
    "\n",
    "samples('Russian', 'RUS')\n",
    "\n",
    "samples('German', 'GER')\n",
    "\n",
    "samples('Spanish', 'SPA')\n",
    "\n",
    "samples('Chinese', 'CHI')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "练习\n",
    "=========\n",
    "\n",
    "-  尝试使用不同 类别->行 数据集, 例如: \n",
    "\n",
    "   -  小说系列 -> 角色名字\n",
    "   -  词性 -> 词语\n",
    "   -  国家 -> 城市\n",
    "\n",
    "-  使用“串起始”标记, 使采样的时候不用给定起始字母\n",
    "-  使用更大和/或更好的网络结构获取更好的结果\n",
    "\n",
    "   -  尝试一下 nn.LSTM 和 nn.GRU 层\n",
    "   -  将这些 RNNs 组合成更高级的网络\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本节完。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}